Threats to Validity

THREATS TO VALIDITY AND

ASSOCIATED CONTROL TECHNIQUES

I. Threats to Internal Validity

Internal validity: Accuracy of the research study in determining that changes in the independent variable CAUSED changes in the dependent variable, as opposed to the possibility that some "confounding" variable may have caused the observed results. Strong internal validity is ESSENTIAL in experimental designs.

Threat

Description

Control Technique

A. Preexisting Subject Variables or Selection Effects
Any factor that creates control and experimental groups that are not equivalent on some important subject variable. Random assignment of subjects to control and experimental groups.

B. History Effects

Changes in the DV that could be due to events that occurred during the study that were not part of the study.

Include a control group in your design.

C. Maturation Effects
Changes in the DV that could be due to the natural maturation or aging of the subjects.
Include a control group in your design.

D. Testing Effects (or practice effects) Changes in the DV that could be due to the fact subjects have been tested on the same activity or given the same test more than once. Include a control group in your design.

E. Regression to the mean

Tendency for subjects who obtain extreme scores on a variable to have less extreme scores on follow-up testing.

Beware of this threat when selecting subjects for inclusion in your study.

F. Instrumentation
Changes in the calibration of a measuring instrument that affects scores on the DV.

Recalibrate instruments often.

G. Diffusion of Treatment

Changes seen in subjects’ responses due to information they received from other subjects in the study.

Controls may not be possible depending on the design of your study.

H. Attrition

The differential loss of subjects during a study. (Those who drop out are likely to be different from those who continue in the study).

No real control.

I. Sequencing (Order Effects)

Effects on a subject’s performance or behavior due to the experiences the subject has had in earlier parts of the study.

Employ counterbalancing techniques.

II. Other Important Threats to Internal Validity:

Subject and Experimenter Effects or Biases

A. Experimenter Bias: Any effect that the "expectations" of the researcher may have had on the measurement and recording of the dependent variable.

Experimenter Effect

Description

Control Technique

Knowledge, expectations, and information that could affect the outcome of the experiment.

Knowledge of which subjects are in the control and experimental groups.

Utilize a single (or double) blind design so that the researcher does not know which subjects are in each group.

Biased measurement (i.e., seeing what the experimenter wants to see).

Use objective (rather than subjective) measurements.

Utilize multiple observers or "research assistants" who are unaware of the hypothesis driving the study. (Check for inter-rater reliability).

B. Subject Effects: Any response by subjects in a study that does not represent the way they would normally behave if not in the study.

Subject Effects

Description

Control Technique

Demand Characteristics

Elements in a study that may cue a subject as to the purpose of the study.

Use deception (i.e., keep the subjects in the dark as to the true purpose of the study).

Placebo Effect or Self-fulfilling Prophecy

An observed improvement in subjects because subjects believe a change will occur.

Provide a placebo for the control group and ensure that all subjects are "blind" as to which group they are in (a single-blind placebo design).

III. Threats to External Validity

Note: Control Techniques are NOT involved

External Validity: The generalizability of the results of the study. The extent to which the results of a particular study can extend to other subjects, times, and settings. This is NOT important in determining cause and effect.

The more NATURAL* the setting, the LARGER the sample, the more REPRESENTATIVE the sample is of the larger population from which it was drawn, and the smaller the standard deviation, the greater the external validity of the study.

REPLICATING a study using different subjects in different settings with slightly different procedures will help you demonstrate external validity of a study.

*Keep in mind that while a "natural" setting increases external validity, it greatly decreases internal validity

Threat	Description	Control Technique
A. Preexisting Subject Variables or Selection Effects	Any factor that creates control and experimental groups that are not equivalent on some important subject variable.	Random assignment of subjects to control and experimental groups.
B. History Effects	Changes in the DV that could be due to events that occurred during the study that were not part of the study.	Include a control group in your design.
C. Maturation Effects	Changes in the DV that could be due to the natural maturation or aging of the subjects.	Include a control group in your design.
D. Testing Effects (or practice effects)	Changes in the DV that could be due to the fact subjects have been tested on the same activity or given the same test more than once.	Include a control group in your design.
E. Regression to the mean	Tendency for subjects who obtain extreme scores on a variable to have less extreme scores on follow-up testing.	Beware of this threat when selecting subjects for inclusion in your study.
F. Instrumentation	Changes in the calibration of a measuring instrument that affects scores on the DV.	Recalibrate instruments often.
G. Diffusion of Treatment	Changes seen in subjects’ responses due to information they received from other subjects in the study.	Controls may not be possible depending on the design of your study.
H. Attrition	The differential loss of subjects during a study. (Those who drop out are likely to be different from those who continue in the study).	No real control.
I. Sequencing (Order Effects)	Effects on a subject’s performance or behavior due to the experiences the subject has had in earlier parts of the study.	Employ counterbalancing techniques.

Experimenter Effect	Description	Control Technique
Knowledge, expectations, and information that could affect the outcome of the experiment.	Knowledge of which subjects are in the control and experimental groups.	Utilize a single (or double) blind design so that the researcher does not know which subjects are in each group.
	Biased measurement (i.e., seeing what the experimenter wants to see).	Use objective (rather than subjective) measurements.
		Utilize multiple observers or "research assistants" who are unaware of the hypothesis driving the study. (Check for inter-rater reliability).

Subject Effects	Description	Control Technique
Demand Characteristics	Elements in a study that may cue a subject as to the purpose of the study.	Use deception (i.e., keep the subjects in the dark as to the true purpose of the study).
Placebo Effect or Self-fulfilling Prophecy	An observed improvement in subjects because subjects believe a change will occur.	Provide a placebo for the control group and ensure that all subjects are "blind" as to which group they are in (a single-blind placebo design).